🐛🎨⚗️Computational backend stability: improvements step 1 #8323
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sanderegg
added
a:director-v2
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #8323 +/- ##
==========================================
- Coverage 87.83% 87.83% -0.01%
==========================================
Files 1945 1945
Lines 75397 75430 +33
Branches 1311 1311
==========================================
+ Hits 66226 66254 +28
- Misses 8776 8781 +5
Partials 395 395
Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
sanderegg
force-pushed
the
computational-backend/improvements-diverse
branch
from
aa2ee3e to
5fa7dac
Compare
sanderegg
changed the title
sanderegg
force-pushed
the
computational-backend/improvements-diverse
branch
3 times, most recently
from
d9c1e1b to
8a082ac
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR improves the stability of the computational backend by addressing timeout issues when retrieving task results from the dask-scheduler. Previously, when task result retrieval timed out, tasks would be immediately marked as FAILED even if they were actually successful. The new implementation introduces a retry mechanism that waits for a configurable period before marking tasks as failed.
Key changes:
- Implements retry logic for task result retrieval with timeout handling
- Adds new configuration setting for maximum wait time when retrieving results
- Refactors task result processing to handle different error types appropriately
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
services/director-v2/src/simcore_service_director_v2/modules/comp_scheduler/_scheduler_dask.py |
Major refactoring of task result processing logic with new error handling and retry mechanisms |
services/director-v2/src/simcore_service_director_v2/core/settings.py |
Adds new configuration setting for maximum wait time when retrieving results |
services/director-v2/src/simcore_service_director_v2/modules/dask_client.py |
Updates logging and removes get_cluster_details method |
services/director-v2/tests/unit/with_dbs/comp_scheduler/test_scheduler_dask.py |
Adds comprehensive test for the new retry behavior |
services/director-v2/tests/unit/with_dbs/comp_scheduler/conftest.py |
Adds fixture for testing the new timeout configuration |
services/director-v2/tests/unit/test_modules_dask_client.py |
Removes test for deleted get_cluster_details functionality |
packages/dask-task-models-library/src/dask_task_models_library/plugins/task_life_cycle_worker_plugin.py |
Adds type assertion for worker instance |
services/director-v2/src/simcore_service_director_v2/core/settings.py
Outdated
Show resolved
Hide resolved
services/director-v2/src/simcore_service_director_v2/modules/comp_scheduler/_scheduler_dask.py
Outdated
Show resolved
Hide resolved
services/director-v2/tests/unit/with_dbs/comp_scheduler/test_scheduler_dask.py
Outdated
Show resolved
Hide resolved
sanderegg
requested review from
JavierGOrdonnez,
bisgaard-itis,
giancarloromeo and
wvangeit
pcrespov
approved these changes
Sep 8, 2025
services/director-v2/src/simcore_service_director_v2/modules/comp_scheduler/_scheduler_dask.py
Show resolved
Hide resolved
services/director-v2/src/simcore_service_director_v2/modules/dask_client.py
Outdated
Show resolved
Hide resolved
sanderegg
force-pushed
the
computational-backend/improvements-diverse
branch
2 times, most recently
from
4af263a to
45e2c7d
Compare
bisgaard-itis
approved these changes
Sep 9, 2025
There was a problem hiding this comment.
Cool, thanks a lot for the effort. I think this PR should close/resolve #7975 and ITISFoundation/osparc-issues#1952.
services/director-v2/src/simcore_service_director_v2/modules/comp_scheduler/_scheduler_dask.py
Outdated
Show resolved
Hide resolved
sanderegg
force-pushed
the
computational-backend/improvements-diverse
branch
2 times, most recently
from
0cbd6d7 to
366ac4b
Compare
GitHK
approved these changes
Sep 9, 2025
…ings.py Co-authored-by: Copilot <[email protected]>
…cheduler_dask.py Co-authored-by: Copilot <[email protected]>
…omp_scheduler/_scheduler_dask.py Co-authored-by: Copilot <[email protected]>
sanderegg
force-pushed
the
computational-backend/improvements-diverse
branch
from
a11cf98 to
19a833b
Compare
|
sanderegg
changed the title
16 tasks
17 tasks
This was referenced Sep 18, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
When the dask-scheduler is not responding in time due to timeouts the director-v2 would set the task as FAILED even when it is actually a success. This triggers this kind of error messages:
This PR changes that behavior by retrying to get the task results several times instead. it also introduces a new constant to define that time which is currently hard-coded (aka no new ENV).
This PR is the first step to improve the computational backend stability since a while. There will be following PRs to tackle:
Related issue/s
How to test
Dev-ops